from sklearn_benchmarks.reporting.hp_match import HPMatchReporting

scikit-learn vs. ONNX Runtime

reporting = HPMatchReporting(against_lib="onnx", config="config.yml")
reporting.make_report()

We assume here there is a perfect match between the hyperparameters of both librairies. For a given set of parameters and a given dataset, we compute the speedup time scikit-learn / time onnx. For instance, a speedup of 2 means that onnx is twice as fast as scikit-learn for a given set of parameters and a given dataset.

KNeighborsClassifier_brute_force

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

All estimators share the following parameters: algorithm=brute.

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.011	0.0	NaN	7.107	0.0	-1	1	NaN	17.251	0.033	0.663	0.001	0.001	See	See
3	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.011	0.0	NaN	7.071	0.0	-1	5	NaN	0.314	0.006	1.000	0.036	0.036	See	See
6	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.011	0.0	NaN	7.099	0.0	1	100	NaN	17.015	0.080	0.882	0.001	0.001	See	See
9	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.011	0.0	NaN	7.069	0.0	-1	100	NaN	0.316	0.006	1.000	0.036	0.036	See	See
12	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.011	0.0	NaN	6.998	0.0	1	5	NaN	3.785	0.009	0.896	0.003	0.003	See	See
15	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.011	0.0	NaN	7.152	0.0	1	1	NaN	0.238	0.004	1.000	0.047	0.047	See	See
18	KNeighborsClassifier_brute_force	fit	100000	100000	2	0.004	0.0	NaN	0.375	0.0	-1	1	NaN	3.803	0.051	0.929	0.001	0.001	See	See
21	KNeighborsClassifier_brute_force	fit	100000	100000	2	0.004	0.0	NaN	0.374	0.0	-1	5	NaN	0.238	0.003	1.000	0.018	0.018	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	KNeighborsClassifier_brute_force	predict	100000	1000	100	1.943	0.146	NaN	0.000	0.002	-1	1	0.663	0.315	0.005	1.000	6.172	6.173	See	See
2	KNeighborsClassifier_brute_force	predict	100000	1	100	0.022	0.003	NaN	0.000	0.022	-1	1	1.000	17.069	0.162	0.757	0.001	0.001	See	See
4	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.772	0.033	NaN	0.000	0.003	-1	5	0.757	17.061	0.014	0.882	0.162	0.162	See	See
5	KNeighborsClassifier_brute_force	predict	100000	1	100	0.023	0.002	NaN	0.000	0.023	-1	5	1.000	0.315	0.005	1.000	0.073	0.073	See	See
7	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.036	0.004	NaN	0.000	0.002	1	100	0.882	0.315	0.005	1.000	6.457	6.458	See	See
8	KNeighborsClassifier_brute_force	predict	100000	1	100	0.020	0.000	NaN	0.000	0.020	1	100	1.000	17.429	0.128	0.757	0.001	0.001	See	See
10	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.832	0.032	NaN	0.000	0.003	-1	100	0.882	17.455	0.155	0.663	0.162	0.162	See	See
11	KNeighborsClassifier_brute_force	predict	100000	1	100	0.024	0.002	NaN	0.000	0.024	-1	100	1.000	0.314	0.007	1.000	0.077	0.077	See	See
13	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.027	0.006	NaN	0.000	0.002	1	5	0.757	0.238	0.004	1.000	8.519	8.521	See	See
14	KNeighborsClassifier_brute_force	predict	100000	1	100	0.020	0.000	NaN	0.000	0.020	1	5	1.000	3.729	0.016	0.922	0.005	0.005	See	See
16	KNeighborsClassifier_brute_force	predict	100000	1000	100	1.166	0.010	NaN	0.001	0.001	1	1	0.663	3.787	0.026	0.929	0.308	0.308	See	See
17	KNeighborsClassifier_brute_force	predict	100000	1	100	0.019	0.001	NaN	0.000	0.019	1	1	1.000	0.238	0.004	1.000	0.080	0.080	See	See
19	KNeighborsClassifier_brute_force	predict	100000	1000	2	1.792	0.022	NaN	0.000	0.002	-1	1	0.896	0.239	0.004	1.000	7.510	7.512	See	See
20	KNeighborsClassifier_brute_force	predict	100000	1	2	0.006	0.004	NaN	0.000	0.006	-1	1	1.000	3.733	0.028	0.922	0.002	0.002	See	See
22	KNeighborsClassifier_brute_force	predict	100000	1000	2	2.918	0.079	NaN	0.000	0.003	-1	5	0.922	3.762	0.007	0.896	0.776	0.776	See	See
23	KNeighborsClassifier_brute_force	predict	100000	1	2	0.006	0.002	NaN	0.000	0.006	-1	5	1.000	0.236	0.004	1.000	0.025	0.025	See	See

KNeighborsClassifier_kd_tree

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

All estimators share the following parameters: algorithm=kd_tree.

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.780	0.017	NaN	0.021	0.0	-1	1	NaN	110.400	0.000	0.929	0.034	0.034	See	See
3	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	4.420	0.043	NaN	0.018	0.0	-1	5	NaN	2.595	0.181	1.000	1.703	1.708	See	See
6	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	4.479	0.059	NaN	0.018	0.0	1	100	NaN	109.299	0.000	0.951	0.041	0.041	See	See
9	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	4.193	0.119	NaN	0.019	0.0	-1	100	NaN	2.593	0.147	1.000	1.617	1.620	See	See
12	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.923	0.068	NaN	0.020	0.0	1	5	NaN	0.049	0.021	0.891	80.558	87.798	See	See
15	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	4.046	0.054	NaN	0.020	0.0	1	1	NaN	0.005	0.000	1.000	786.282	786.576	See	See
18	KNeighborsClassifier_kd_tree	fit	1000	1000	2	0.001	0.001	NaN	0.021	0.0	-1	1	NaN	0.062	0.002	0.894	0.012	0.012	See	See
21	KNeighborsClassifier_kd_tree	fit	1000	1000	2	0.001	0.000	NaN	0.031	0.0	-1	5	NaN	0.005	0.000	1.000	0.102	0.102	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.864	1.007	NaN	0.000	0.001	-1	1	0.929	2.621	0.201	1.000	0.330	0.330	See	See
2	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.001	NaN	0.000	0.003	-1	1	1.000	109.714	0.000	0.946	0.000	0.000	See	See
4	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.128	0.339	NaN	0.000	0.001	-1	5	0.946	108.848	0.000	0.951	0.010	0.010	See	See
5	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.001	NaN	0.000	0.003	-1	5	1.000	2.590	0.154	1.000	0.001	0.001	See	See
7	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	5.869	0.340	NaN	0.000	0.006	1	100	0.951	2.599	0.179	1.000	2.258	2.264	See	See
8	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.001	NaN	0.000	0.003	1	100	1.000	110.284	0.000	0.946	0.000	0.000	See	See
10	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	3.084	0.181	NaN	0.000	0.003	-1	100	0.951	109.185	0.000	0.929	0.028	0.028	See	See
11	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.005	0.001	NaN	0.000	0.005	-1	100	1.000	2.609	0.172	1.000	0.002	0.002	See	See
13	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.792	0.289	NaN	0.000	0.002	1	5	0.946	0.005	0.000	1.000	353.501	353.723	See	See
14	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.002	0.000	NaN	0.000	0.002	1	5	1.000	0.040	0.001	0.911	0.039	0.039	See	See
16	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.969	0.261	NaN	0.000	0.001	1	1	0.929	0.062	0.002	0.894	15.690	15.699	See	See
17	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.001	0.000	NaN	0.000	0.001	1	1	1.000	0.005	0.000	1.000	0.188	0.188	See	See
19	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.025	0.010	NaN	0.001	0.000	-1	1	0.891	0.005	0.000	1.000	4.904	4.905	See	See
20	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.002	0.000	NaN	0.000	0.002	-1	1	1.000	0.039	0.000	0.911	0.052	0.052	See	See
22	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.025	0.001	NaN	0.001	0.000	-1	5	0.911	0.039	0.000	0.891	0.635	0.635	See	See
23	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.002	0.000	NaN	0.000	0.002	-1	5	1.000	0.005	0.000	1.000	0.409	0.409	See	See

HistGradientBoostingClassifier_best

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

All estimators share the following parameters: learning_rate=0.01, n_iter_no_change=10.0, max_leaf_nodes=100.0, max_bins=255.0, min_samples_leaf=100.0, max_iter=300.0.

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	HistGradientBoostingClassifier_best	fit	100000	100000	100	95.929	0.0	300	0.001	0.001	NaN	0.494	0.021	0.824	194.193	194.371	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	HistGradientBoostingClassifier_best	predict	100000	1000	100	0.11	0.002	300	0.007	0.0	0.824	0.406	0.006	1.0	0.271	0.271	See	See